fix(ci): the invisible-character gate never matched anything - #193

🏁 Script executed:

#!/bin/bash
set -e
printf '%s\n' '--- workflow context ---'
sed -n '95,155p' .github/workflows/dogfood-gate.yml
printf '%s\n' '--- relevant commands and variables ---'
rg -n -C 3 'PATTERNS|empty-lint-results|grep -P|grep -a|BOM|bom|FEFF' .github/workflows/dogfood-gate.yml

Repository: hyperpolymath/verisimiser

Length of output: 5095

🏁 Script executed:

#!/bin/bash
set -e
printf '%s\n' '--- GNU grep version ---'
grep --version | head -n 1
PATTERNS='\x00|[\x01-\x08\x0B\x0C\x0E-\x1F]|\x{a0}|\x{ad}|\x{200b}|\x{200c}|\x{200d}|\x{200e}|\x{200f}|\x{202a}|\x{202b}|\x{202c}|\x{202d}|\x{202e}|\x{2060}|\x{feff}'
printf '%s\n' '--- leading UTF-8 BOM ---'
if grep -aPq "$PATTERNS" <(printf '\357\273\277source\n'); then echo MATCH; else echo NO_MATCH; fi
printf '%s\n' '--- embedded UTF-8 BOM ---'
if grep -aPq "$PATTERNS" <(printf 'source\357\273\277text\n'); then echo MATCH; else echo NO_MATCH; fi
printf '%s\n' '--- ordinary text ---'
if grep -aPq "$PATTERNS" <(printf 'source\n'); then echo MATCH; else echo NO_MATCH; fi

Repository: hyperpolymath/verisimiser

Length of output: 496

🏁 Script executed:

#!/bin/bash
set -e
printf '%s\n' '--- locale ---'
locale
PATTERNS='\x00|[\x01-\x08\x0B\x0C\x0E-\x1F]|\x{a0}|\x{ad}|\x{200b}|\x{200c}|\x{200d}|\x{200e}|\x{200f}|\x{202a}|\x{202b}|\x{202c}|\x{202d}|\x{202e}|\x{2060}|\x{feff}'
for lang in C C.UTF-8 en_US.UTF-8; do
  printf '%s\n' "--- LANG=$lang ---"
  if LANG="$lang" LC_ALL="$lang" grep -aPq "$PATTERNS" <(printf '\357\273\277source\n') 2>/tmp/grep-error; then
    echo MATCH
  else
    status=$?
    echo "STATUS=$status"
  fi
  if [ -s /tmp/grep-error ]; then cat /tmp/grep-error; fi
done

Repository: hyperpolymath/verisimiser

Length of output: 222

🏁 Script executed:

#!/bin/bash
set +e
printf '%s\n' '--- environment ---'
env | grep -E '^(LANG|LC_)' || true
grep --version | head -n 1
for lang in C C.UTF-8 en_US.UTF-8; do
  printf '%s\n' "--- LANG=$lang ---"
  LANG="$lang" LC_ALL="$lang" grep -aPq '\x{feff}' <(printf '\357\273\277source\n') 2>/tmp/grep-error
  echo "leading_status=$?"
  if [ -s /tmp/grep-error ]; then cat /tmp/grep-error; fi
  LANG="$lang" LC_ALL="$lang" grep -aPq '\x{feff}' <(printf 'source\357\273\277text\n') 2>/tmp/grep-error
  echo "embedded_status=$?"
  if [ -s /tmp/grep-error ]; then cat /tmp/grep-error; fi
done

Repository: hyperpolymath/verisimiser

Length of output: 750

🌐 Web query:

GNU grep -P "\x{feff}" "character code point value in" PCRE2 UTF support

💡 Result:

The character code point value for the Byte Order Mark (BOM) in Unicode is U+FEFF [1][2]. In the context of PCRE2 and tools that utilize it—such as GNU grep (when using the -P or --perl-regexp flag) or pcre2grep—the handling of this character follows standard Unicode and regular expression rules: 1. Code Point Value: U+FEFF represents the "ZERO WIDTH NO-BREAK SPACE" character, which is repurposed as a Byte Order Mark (BOM) when placed at the beginning of a text stream [1][2]. 2. Representation in PCRE2: In regular expressions, this code point is commonly represented using the escape sequence \x{feff} [2]. 3. Behavior with PCRE2: When PCRE2 is in UTF mode (which is typically enabled in GNU grep and pcre2grep when processing UTF-8 data), \x{feff} is treated as a literal character matching the Unicode code point U+FEFF [1]. 4. UTF-8 BOM vs. Literal Character: It is important to distinguish between the file-level BOM (which is a sequence of bytes, EF BB BF in UTF-8) and the character U+FEFF within a string [2]. Most grep implementations do not automatically ignore or strip the BOM from the input stream; if a file starts with a UTF-8 BOM, that BOM is present as the first three bytes of the file [3]. If you search for \x{feff} in a file containing a UTF-8 BOM, the regex engine will generally not match it unless the regex engine is specifically configured to interpret those bytes as the U+FEFF character, which is usually not the case with raw byte-stream grep operations [4]. In practice, if you need to match a UTF-8 BOM at the start of a file using regex, it is often more reliable to match the raw byte sequence (\xEF\xBB\xBF) rather than relying on the U+FEFF character interpretation, as the latter depends on the engine's encoding awareness and the presence of the BOM in the input [4][5].

Citations:

• 1: https://web.archive.org/web/20210502091429/http:/www.unicode.org/faq/utf_bom.html
• 2: https://unicodefyi.com/glossary/byte-order-mark/
• 3: https://forum.dbpoweramp.com/forum/dbpoweramp/cd-ripper/44456-utf-16-le-bom-log-and-utf-8-cue-some-files-grep-can-t-see
• 4: https://til.simonwillison.net/bash/finding-bom-csv-files-with-ripgrep
• 5: https://unix.stackexchange.com/questions/381230/how-can-i-remove-the-bom-from-a-utf-8-file